Probability Theorems: Multiplication Law and Total Probability

Multiplication Theorem on Probability (for Dependent and Independent Events)

Purpose

The **Multiplication Theorem of Probability**, also known as the Multiplication Rule, provides a method to calculate the probability that two or more events will **both occur**. This is the probability of the intersection of the events, often denoted as $P(A \cap B)$ for two events A and B, or $P(A \text{ and } B)$.

The general form of the multiplication rule is derived directly from the definition of conditional probability and is applicable to any two events, whether they are dependent or independent.

General Multiplication Rule (for any two events)

Recall the definition of conditional probability of event A given event B, $P(A|B) = \frac{P(A \cap B)}{P(B)}$, provided $P(B) > 0$. Similarly, $P(B|A) = \frac{P(A \cap B)}{P(A)}$, provided $P(A) > 0$.

We can rearrange these definitions to solve for $P(A \cap B)$.

Multiplying the first equation by $P(B)$ gives:

Multiplying the second equation by $P(A)$ gives:

These are the two forms of the general multiplication rule. They state that the probability that both events A and B will occur is equal to the probability of one event occurring multiplied by the conditional probability of the other event occurring given that the first event has already occurred.

These formulas are always true, provided the conditioning event has a non-zero probability.

Multiplication Rule for Independent Events

Two events A and B are defined as **independent** if the occurrence of one event does not influence or change the probability of the other event occurring. Mathematically, A and B are independent if and only if $P(A|B) = P(A)$ (if $P(B)>0$) or $P(B|A) = P(B)$ (if $P(A)>0$).

If A and B are independent events, we can substitute $P(B|A) = P(B)$ into the general multiplication rule $P(A \cap B) = P(A) \cdot P(B|A)$. This yields the special multiplication rule for independent events:

This means that the probability of two independent events both occurring is simply the product of their individual probabilities.

Extension to Multiple Events

The multiplication rule can be extended to find the probability of the intersection of three or more events.

For three events A, B, and C, the probability that all three occur is given by:

This formula assumes that $P(A)>0$ and $P(A \cap B)>0$. It states that the probability of A, B, and C all occurring is the probability of A, times the probability of B given A, times the probability of C given that both A and B have occurred.

If the three events A, B, and C are **mutually independent** (meaning the outcome of any one or any combination of them does not affect the probability of the remaining events), then the conditional probabilities simplify:

• $P(B|A) = P(B)$
• $P(C|A \cap B) = P(C)$

In this case, the multiplication rule for three mutually independent events simplifies to:

This extends to any number of mutually independent events: the probability of their intersection is the product of their individual probabilities.

Example

Answer:

Independent Events: Definition and Test

Definition

In probability, two events, $A$ and $B$, associated with the same random experiment are defined as **statistically independent** (or simply independent) if the occurrence or non-occurrence of one event does **not affect** the probability of the occurrence of the other event. Knowing whether one event happened or not does not change the likelihood of the other event happening.

Formally, event $A$ is independent of event $B$ if and only if:

This definition states that the conditional probability of A given B is the same as the marginal (unconditional) probability of A. Similarly, event $B$ is independent of event $A$ if and only if:

If $P(A)>0$ and $P(B)>0$, then $P(A|B)=P(A)$ if and only if $P(B|A)=P(B)$. The relationship of independence is symmetric.

If two events are not independent, they are called **dependent events**. For dependent events, the probability of one event occurring changes based on whether the other event has occurred (i.e., $P(A|B) \neq P(A)$ or $P(B|A) \neq P(B)$).

By convention, if $P(A)=0$ or $P(B)=0$, the events A and B are generally considered independent.

Test for Independence (Multiplication Rule Test)

The definition of independence using conditional probability ($P(A|B)=P(A)$) is useful conceptually, but it requires calculating a conditional probability. A more practical and widely used way to test if two events A and B are independent is by checking if their joint probability (the probability of their intersection) is equal to the product of their individual probabilities. This is derived from the multiplication rule.

Recall the general multiplication rule: $P(A \cap B) = P(A) \cdot P(B|A)$. If A and B are independent, $P(B|A) = P(B)$. Substituting this into the general rule gives $P(A \cap B) = P(A) \cdot P(B)$.

Conversely, if $P(A \cap B) = P(A) \cdot P(B)$, then (assuming $P(A)>0$) $P(B|A) = \frac{P(A \cap B)}{P(A)} = \frac{P(A)P(B)}{P(A)} = P(B)$, confirming independence.

Therefore, events A and B are independent **if and only if**:

This equation serves as the primary test for statistical independence between two events.

Procedure to Test for Independence using the Multiplication Rule:

1. Calculate the probability of event A, $P(A)$.
2. Calculate the probability of event B, $P(B)$.
3. Calculate the probability that both events A and B occur simultaneously, $P(A \cap B)$.
4. Calculate the product of the individual probabilities: $P(A) \cdot P(B)$.
5. Compare the results from step 3 and step 4:
   • If $P(A \cap B) = P(A) \cdot P(B)$, the events A and B are **independent**.
   • If $P(A \cap B) \neq P(A) \cdot P(B)$, the events A and B are **dependent**.

Independence of More Than Two Events

For more than two events, the concept of independence needs careful definition. For three events A, B, and C to be **mutually independent**, it is not sufficient for them to be pairwise independent. Mutual independence requires that the probability of the intersection of any subset of these events is equal to the product of their individual probabilities.

For three events A, B, and C to be mutually independent, all of the following conditions must be satisfied:

• $P(A \cap B) = P(A) P(B)$ (A and B are independent)
• $P(A \cap C) = P(A) P(C)$ (A and C are independent)
• $P(B \cap C) = P(B) P(C)$ (B and C are independent)
• $P(A \cap B \cap C) = P(A) P(B) P(C)$ (All three together are independent)

If only the first three conditions hold, the events are said to be **pairwise independent**, but not necessarily mutually independent.

Example

Answer:

Partition of Sample Space

Definition

In probability theory, a set of events $E_1, E_2, \dots, E_n$ is said to form a **partition** of the sample space $S$ if they collectively cover the entire sample space without overlapping. Formally, a collection of events $\{E_1, E_2, \dots, E_n\}$ constitutes a partition of $S$ if the following three conditions are satisfied:

1. Mutually Exclusive (Pairwise Disjoint):

   Any two distinct events in the collection are mutually exclusive. This means that if one event occurs, none of the other events in the collection can occur simultaneously in a single trial of the experiment.

   $$E_i \cap E_j = \phi \quad \text{for all } i \neq j, \text{ where } i, j \in \{1, 2, \dots, n\}$$

   ... (1)

2. Exhaustive:

   The union of all the events in the collection is equal to the entire sample space. This means that at least one of these events must occur whenever the experiment is performed; there are no outcomes in $S$ that are not included in at least one of the $E_i$'s.

   $$E_1 \cup E_2 \cup \dots \cup E_n = \bigcup\limits_{i=1}^{n} E_i = S$$

   ... (2)

3. Non-Zero Probability (Standard Assumption for many theorems):

   Each event in the partition is usually assumed to have a positive probability of occurring.

   $$P(E_i) > 0 \quad \text{for all } i \in \{1, 2, \dots, n\}$$

   ... (3)

   While conditions 1 and 2 strictly define a partition, condition 3 is often added as a requirement for partitions used in theorems like the Law of Total Probability and Bayes' Theorem, to avoid division by zero.

Visually, a partition divides the sample space $S$ into a set of non-overlapping regions that together fill up $S$.

Examples of Partitions

• Rolling a standard six-sided die ($S=\{1, 2, 3, 4, 5, 6\}$):

  • Let $E_1=\{1, 2\}$, $E_2=\{3, 4\}$, $E_3=\{5, 6\}$. Are they mutually exclusive? Yes, $E_1 \cap E_2 = \phi$, $E_1 \cap E_3 = \phi$, $E_2 \cap E_3 = \phi$. Are they exhaustive? Yes, $E_1 \cup E_2 \cup E_3 = \{1, 2, 3, 4, 5, 6\} = S$. Do they have positive probability? Yes, $P(E_1)=2/6$, $P(E_2)=2/6$, $P(E_3)=2/6$. So, $\{E_1, E_2, E_3\}$ form a partition of $S$.
  • Let $E_{odd}=\{1, 3, 5\}$ (getting an odd number) and $E_{even}=\{2, 4, 6\}$ (getting an even number). Are they mutually exclusive? Yes, $E_{odd} \cap E_{even} = \phi$. Are they exhaustive? Yes, $E_{odd} \cup E_{even} = \{1, 3, 5, 2, 4, 6\} = S$. Do they have positive probability? Yes, $P(E_{odd})=3/6$, $P(E_{even})=3/6$. So, $\{E_{odd}, E_{even}\}$ form a partition of $S$.
  • Let $A=\{1, 2, 3\}$ and $B=\{3, 4, 5, 6\}$. Are they exhaustive? Yes, $A \cup B = S$. Are they mutually exclusive? No, $A \cap B = \{3\} \neq \phi$. So, $\{A, B\}$ do **not** form a partition.

• Any Event and its Complement:

  For any event $A$ in the sample space $S$, the set $\{A, A'\}$ forms a partition of the sample space, provided $0 < P(A) < 1$. If $P(A)=0$ or $P(A)=1$, technically one of the events in the partition would have zero probability.

  • Are A and A' mutually exclusive? Yes, $A \cap A' = \phi$.
  • Are A and A' exhaustive? Yes, $A \cup A' = S$.

The concept of partitioning the sample space into a set of mutually exclusive and exhaustive events is fundamental to solving many probability problems, especially those involving sequential events or multiple stages, as it forms the basis for the Law of Total Probability and Bayes' Theorem.

The Law of Total Probability

Statement

The **Law of Total Probability** is a fundamental theorem that relates the probability of an event to the conditional probabilities of that event given a partition of the sample space. It is used to calculate the overall probability of an event when the conditions under which it can occur are divided into several mutually exclusive and exhaustive scenarios.

Let $\{E_1, E_2, \dots, E_n\}$ be a partition of the sample space $S$. This means the events $E_i$ are pairwise mutually exclusive ($E_i \cap E_j = \phi$ for $i \neq j$), their union is the entire sample space ($\bigcup\limits_{i=1}^{n} E_i = S$), and we assume $P(E_i) > 0$ for all $i=1, \dots, n$.

Let $A$ be any event associated with the sample space $S$. The Law of Total Probability states that the probability of event A can be calculated as the sum of the conditional probabilities of A given each event in the partition ($P(A|E_i)$), weighted by the probability of each event in the partition ($P(E_i)$).

Formula for the Law of Total Probability:

Using summation notation, this is:

Derivation and Intuition

The Law of Total Probability can be derived using the properties of partitions and the multiplication rule:

Since $\{E_1, E_2, \dots, E_n\}$ is a partition of $S$, any event $A$ within $S$ can be expressed as the union of its intersections with each event in the partition.

Because the events $E_i$ are pairwise mutually exclusive, their intersections with event A, $(A \cap E_i)$, are also pairwise mutually exclusive. For any $i \neq j$, $(A \cap E_i) \cap (A \cap E_j) = A \cap A \cap E_i \cap E_j = A \cap \phi = \phi$.

Since the events $(A \cap E_1), (A \cap E_2), \dots, (A \cap E_n)$ are mutually exclusive, by Axiom 3 (Additivity of probability for mutually exclusive events), the probability of their union is the sum of their individual probabilities:

Now, recall the general Multiplication Rule (Formula 2 from Section I1): $P(A \cap E_i) = P(E_i) P(A|E_i)$ (assuming $P(E_i)>0$). Substitute this into each term of the sum in equation (iv):

This completes the derivation of the Law of Total Probability.

The intuition is that the overall probability of event A is the weighted average of the conditional probabilities of A under each scenario $E_i$, where the weights are the probabilities of those scenarios occurring.

Purpose and Application

The Law of Total Probability is particularly useful in multi-stage experiments or when calculating the probability of an event $A$ that can occur under several different, mutually exclusive circumstances or conditions. You use this law when:

• You want to find the probability of a specific event $A$.
• You can identify a set of mutually exclusive and exhaustive events ($E_i$) that cover all possible initial situations or stages of the experiment.
• You know the probability of each of these initial situations $E_i$ occurring ($P(E_i)$).
• You know the conditional probability of event $A$ occurring given that each of the initial situations $E_i$ has occurred ($P(A|E_i)$).

It provides a way to break down the calculation of a complex probability $P(A)$ into simpler calculations involving conditional probabilities related to a partition of the sample space. It is a fundamental stepping stone to understanding and applying Bayes' Theorem.

Example

Answer: